On Complexity and Emergence

Numerous definitions for complexity have been proposed over the last half century, with little consensus achieved on how to use the term. A definition of complexity is supplied here that is closely related to the Kolmogorov Complexity and Shannon Entropy measures widely used as complexity measures, yet addresses a number of concerns raised against these measures. However, the price of doing this is to introduce context dependence into the definition of complexity. It is argued that such context dependence is an inherent property of complexity, and related concepts such as entropy and emergence. Scientists are uncomfortable with such context dependence, which smacks of subjectivity, and this is perhaps the reason why little agreement has been found on the meaning of these terms. 1. The problem of Complexity In the last 15 years, the study of Complex Systems has emerged as a recognised field in its own right, although a good definition of what a complex system actually is has eluded formulation. Attempts to formalise the concept of complexity go back even further, to Shannon’s inception of Information Theory Shannon, 1949. A good survey of the tortuous path the study of complexity has followed is provided in Edmonds (1949). Of particular importance to this paper is the concept of Kolmogorov Complexity (also known as Algorithmic Information Complexity) introduced independently by Kolmogorov (1965), Chaitin (1966) and Solomonoff (1964). Given a particular universal Turing Machine1 (UTM) U , the Kolmogorov complexity of a string of characters (ie a description) is the length of the shortest program running on U that generates the description. There are two main problems with Kolmogorov complexity: 1. The dependence on U , as there is no unique way of specifying this. The Invariance theoremLi and Vitányi, 1997, Thm 2.1.1 guarantees that any two UTMs U and V will agree 1A Turing Machine is a formal model of a digital computer Received: 14 Feb 2001 c Copyright 2001 Accepted: Pending – 1 – http://www.complexity.org.au/vol09/standi09/ Complexity International Volume 09 on the complexity of a string x up to a constant independent of x. However, for any two descriptions x and y, there will be two machines U and V disagreeing on whether x is more complex that y, or vice-versa. 2. Random sequences have maximum complexity, as by definition a random sequence can have no generating algorithm shorter than simply listing the sequence. As Gell-Mann (1964) points out, this contradicts the notion that random sequences should contain no information. The first problem of what reference machine to choose is a symptom of context dependence of complexity. Given a description x, any value of complexity can be chosen for it by choosing an appropriate reference machine. It would seem that complexity is in the eye of the beholder Emmeche, 1994. Is complexity completely subjective? Is everything lost? Rather than trying to hide this context dependence, I would prefer to make it a feature. Instead of asserting complexity is a property of some system, it is a property of descriptions (which may or may not be about a system). There must also be an interpreter of these descriptions that can answer the question of whether two descriptions are equivalent or not. Consider Shakespeare’s Romeo and Juliet. In Act II,ii, line 58, Juliet says “My ears have yet not drunk a hundred words”. If we change the word “drunk” to “heard”, your average theatre goer will not spot the difference. Perhaps the only one to notice would be a professional actor who has played the scene many times. Therefore the different texts differing by the single word “drunk/heard” in this scene are considered equivalent by our hypothetical theatre goer. There will be a whole equivalence class of texts that would be considered to be Romeo and Juliet by our theatre goer. Once the set of all possible descriptions are given (strings of letters on a page, base pairs in a genome or bits on a computer harddisk for example), and an equivalence class between descriptions given, then one can apply the Shannon entropy formula to determine the complexity of that description, under that interpretation: C(x) = lim s!1 s log2N log2 !(s; x) (1) where C(x) is the complexity (measured in bits), N the size of the alphabet used to encode the description and !(s; x) the size of the class of all descriptions equivalent to x and of length less than s. We assume that the interpreter of a description is able to determine where a description finishes, so that a description y of length `(y) is equivalent to all N s `(y) length s descriptions having y as a prefix. If we choose our description set to be all bitstrings, and our equivalence class to be all bitstrings that produce the same output when executed by a universal Turing machine U , then